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ALIGNING INSTRUCTION AND ASSESSMENT WITH GAME 
AND SIMULATION DESIGN 

Richard Wainess, Alan Koenig, and Deirdre Kerr 
CRESST/University of California, Los Angeles 

Abstract 

Effective design of training-related games (games for training and/or assessment) 
requires synergy between the mechanisms for delivering instructional content and the 
mechanisms for learning game play and game functionality (Becker, 2006). The learning 
domain must be embedded as a core game mechanic: that is, the game cannot be 
advanced or won without utilization of the domain being taught or assessed (Fisch, 

2005). To address these issues, we created two interconnected models central to the 
design and development of training -related games: 1) a Game Play Model comprising the 
key components of a game and 2) a Player Interaction Framework defining how players 
interact with information in a game. The model and framework help to optimize the 
design and development process by providing a shared set of categories for organizing 
domain and game instruction. They also provide a lens through which comparisons of 
instructional methods and strategics can be made across games. Using the Game Play 
Model and Player Interaction Framework, we analyzed 34 games (24 popular commercial 
video games and 10 commercial video games used by the military). Results of the 
analyses indicate that while the two game types were si mi lar in the amount of instruction 
devoted to introducing the various components of the Game Play Model, the delivery 
mechanisms (the Player Interaction Framework) differed in some key areas. In particular, 
the military games did not provide enough direct instruction and relied too much on the 
player to actively seek out information. The Game Play Model and the Player Interaction 
Framework represent important components in the design and development process for 
training games and they also provide a useful lens for the examination of the 
effectiveness of instruction within training games. 

Why Game and Domain Instruction Should Be Aligned 

There is a strong consensus in the research community that learning outcomes from 
games are affected by the instructional methods and strategies employed in the games and 
not by the games in and of themselves (e.g., Garris, Ahlers, & Driskell, 2002; Leemkuil, de 
Jong, de Hoog, and Christoph, 2003; Thiagarajan, 1998; Wolfe, 1997). More recently, 
researchers have argued that learning outcomes are dependent on how well the instruction is 
integrated into the game. Egenfeldt-Nielsen (2006) commented that instructional information 
should be a natural part of the game dynamics and must be necessary for success in the game. 
In other words, the act of playing and succeeding at a game should draw directly on the 
knowledge or skills the game was designed to teach (Fisch, 2005). This means there must be 
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integration with the game’s mechanics and in particular, with the game’s core mechanics (the 
actions a player must successfully perform to reach the game’s goals). 

Wainess and colleagues (Wainess, Iseli, Koenig, Choi et al, 2010; Wainess, Koenig, & 
Kerr, in preparation) have examined whether the instructional methods and strategies utilized 
when instructing others how to play entertainment and educational games are the same as 
those used in teaching educational content. Results of the research suggest they are the 
same — with one minor difference, regarding the amount of support provided for discovery 
learning (Wainess et al, 2010). Wainess and colleagues proposed that by understanding 
when, where, and how effective instructional methods and strategies are used for learning to 
play a game and by aligning the needs of teaching an instructional domain with the needs of 
teaching game play, two key benefits can be achieved. First is an alignment of game 
instruction and learning domain instruction. A second benefit is a reduced cognitive load. 
The alignment of game instruction and domain instruction supports the growing argument for 
integration of game and domain learning (Egenfeldt-Nielsen, 2006; Fisch, 2005). Cognitive 
load (the amount of mental activity imposed on work memory at an instance in time; 
Chalmers, 2003, Sweller & Chandler, 1994) can be reduced by limiting the amount of 
germane cognitive load provided in the game (Ayers, 2006). Germaine cognitive load is the 
cognitive load required by the methods used for presenting new knowledge to a learner 
(Renkl & Atkinson, 2003). Blending learning the game with learning the content can lead to 
efficient and effective instruction, as it uses one set of instructions (germane cognitive load) 
to teach two sets of knowledge: game and learning domain. 

The first step to achieving the goals of integration and reduced cognitive load is to map 
instructional methods and strategies and related constructs to how games teach game 
mechanics. The hypothesis is that once we understand how games utilize instructional 
methods, instructional strategies, and other factors that teach how to play a game, we can 
map those methods and strategies to methods and strategies applicable to teaching specific 
instructional content (e.g., teaching fractions). By aligning the methods and strategies for 
teaching game mechanics, tactics, and strategies with the methods and strategies for teaching 
and practicing instructional content, we propose that we can better control the burdens placed 
on working memory, thereby improving the learning process and ultimately learning 
outcomes. Figure 1 shows how game and instructional domain are linked. This paper is 
focused primarily on the left column (the instructional methods). 
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INSTRUCTION 




Figure 1. How games and instructional domain 
are linked. 



By beginning with a learning goal (top of Figure 1) and determining effective 
instructional methods for achieving those goals, a game developer can link those learning 
domain instructional methods to equivalent instructional methods within games that are 
utilized for teaching particular game mechanics and ultimately core mechanics. In other 
words, you would determine the content and how it could effectively be taught. Then, you 
would determine which type of games mechanics use those same instructional methods and 
build a game using those mechanics. You would also want to ensure that the game mechanics 
that are most closely aligned with the learning domain are implemented as core mechanics, 
so that use of the learning domain becomes critical to game success. 

Two Components Important to Improving Instructional Games and Simulations 

The University of California Los Angeles’ National Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) has established a methodology for creating games 
for learning, based on achieving the goals of integrating game instruction and domain 
instruction and of embedding the learning domain into the core game mechanics. Figure 2 
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shows the iterative game development methodology. See Wainess & Koenig (2010) for a 
description of the process. 




Figure 2. Learning game design and development 
process. 



Two components in CRESST’s design and development methodology (Figure 1) are 
relevant to this paper. 

1. The Player Interaction Framework 

2. The Game Play Model 

Player Interaction Framework 

Figure 3 shows the Player Interaction Framework. The framework depicts how a player 
interacts with information (instruction and assessment) in a game space. More specifically, it 
illustrates the ways in which information (domain content and instruction, and assessment 
items) is either presented to the player or how the player can seek out information. 

Presentation objects refer to instruction directly presented to the player (e.g., a dialog 
box with instruction). 

Background objects refer to information or instruction that is covertly integrated into 
the environment and requires the player to actively pursue (seek out) the information (e.g., 
relevant information on a poster on the wall in a hallway alongside other posters that may not 
contain relevant information). 
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Figure 3. Player Interaction Framework. 



Person-to-object interaction and person-to-character/player interaction are similar. 
The object or person is highlighted (visually emphasized), cueing the player that the object or 
person is important and should be interacted with (e.g., a “glowing” object on a table). 

Storage and workshop objects refer to a group of functions that would typically be 
separated from the main game space in order to view items collected (resources), manipulate 
or combine resources, or search for additional information or resources. 

Game Play Model 

The Game Play Model (Figure 4) illustrates the relationships among the components of 
a game and is linked to the instruction by the player interaction framework. For a complete 
discussion of this model, including how it integrates with the player interaction framework, 
see Wainess & Koenig (in preparation). 



PLAYER ENTRANCE 




Figure 4. Game Play Model. 



According to the Game Play Model, a player enters a game space governed by rules. 
The player is given a goal. The player possesses pre-existing affordances (prior knowledge, 
prior skills, self-beliefs, attitudes, etc.) that may or may not be beneficial in the game. The 
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player may also bring with or act upon his or her own goals. The player is represented by the 
term ajforclances , as the affordances represent the current abilities of a player in the game. 
The player’s actions may cause effects in the game that may a) alter the player’s affordance 
and/or b) alter the game space. Changes to the game space may change the game goal(s) 
and/or the player’s affordances, and may eventually cause additional effects, which might in 
turn alter the player’s affordances and/or the game space. 

The Player Interaction Framework (Figure 3) and the Game Play Model (Figure 4) are 
the basis for the research and recommendations in this study. 

Methodology 

This paper utilizes a descriptive study with coding of learning events and comparison 
of two game types: commercial video games used by the military and popular commercial 
video games not used by the military. Ten commercial video games used by the military 
(Table 1) were analyzed and coded for how game features were taught. 

Table 1 



Commercial Video Games Used by the Military 



Game 


Platform 


Branch 


Air Force Delta Storm 


XBOX 


Air force 


Arma II: Ultimate Military Simulator 


PC 


Marines 


Battlefield 1942 


PC 


Army 


(Jane's) Fleet Command 


PC 


Navy 


Steel Beasts 


PC 


Army 


Medal of Honor: Frontline 


PS2 


Marines 


Medal of Honor: War Chest 


PC 


Marines 


Operation Flashpoint: Dragon Rising 


PC 


Army 


SOCOM: US Navy Seals 


PS2 


Navy 


Soldier of Fortune: Payback 


PC 


Marines 



Introduction of three game features (game mechanics, game controls, and game 
interface) were coded for interaction method (Figure 3) and game play component (Figure 4), 
as well as for a wide range of instructional features, including instructional method and 
strategy, type of feedback, and metacognitive support. 
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Table 2 



Popular Commercial Video Games Coded and 
Analyzed for the Present Study 

Name of game 

Animal Crossing 

BATTLEFIELD2 Modern Combat 

Call of Duty 2 

Dark Void Demo 

Elite Beat Agents 

Final Fantasy Tactics 

FolkLore 

Genji: Days of the Blade 

Ghost Recon Advance War Fighter 

Halo 

Heavenly Sword 

Hot Shots Golf: Out of Bounds 

Kill Zone 2 

Kung Fu Panda 

Little Big Planet 

Loco Roco 2 

Motor Storm 

Perfect Dark Zero 

Pokemon Pearl 

Ratchet & Clank: Futute: Tools of Destruction 
Resistance 2 

Spyro: The Eternal Night 
Uncharted 2: Among Thieves 
Worms: Open Warfare 



Twenty-four popular commercial video games (Table 2) were also analyzed and coded. 
Codes for the two game types were examined and compared, using descriptive statistics, 
including frequency and percentages. 

According to Wainess et al. (2010), game mechanics are the actions a player can do, 
governed by the rules of the game. Game controls are the mechanisms by which a player can 
interact with a game. Game controls for video games include button presses, mouse clicks, 
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joy stick movement, and even body movement. Game interface refers to the elements or tools 
a player can see, that allow the player to make choices related to a game (load/save, select a 
particular weapon), view the player’s current state (e.g. a health bar, number of bullets 
remaining in a clip), and view the world’s state (e.g, access maps, view a radar showing 
enemy locations). The game interface is the link between the player and the game world. 

Instructional methods are external supports for metacognitive processes. For example, 
use of analogies is an instructional method. Use of worked examples is another instructional 
method. Instructional strategies are approaches to learning and can benefit from the 
inclusion of instructional methods. Group exercises and lectures are examples of instructional 
strategies. Both group exercises and lectures (instructional strategies) can utilize analogies 
and worked examples (instructional methods). 

Sample 

Two groups of games were examined in this study: popular commercial video games 
and commercial video games utilized by the military. In some cases, a commercial game 
utilized by the military was also a popular commercial video game (e.g., SOCOM: U.S. Navy 
Seals). In those instances, the game was included in the military video game analyses and not 
in the commercial video game analyses. 

Commercial video games utilized by the military were selected from a list of 66 games 
identified by the Department of Defense Game Developer’s Community as having been used 
for military training (http://adlcommunity.net/mod/data/view.php?id=663). Of the 66 games 
listed on the website, only 13 were selected for this study. We were unable to locate many of 
the games, due to the age of the game. Others games were not included because of the cost of 
the game (e.g., base price for Virtual Battle Space II was $1,500). Others were not 
obtainable, such as Avant Guard, by the Air Force Research Lab, Human Effectiveness 
Directorate. 

Training the Coders 

Two researchers were trained on how to analyze and code video games. Each of the 
games was analyzed and coded by only one researcher. To validate the reliability of using a 
single coder, two of the games were examined independently by both coders. The coders then 
reviewed their codes with each other and marked all entries as 1 (agreed to) or 0 (not agreed 
to). The two games analyzed by both coders were Call of Duty 2 (a popular commercial 
game) and SOCOM: US Navy Seals (a commercial video game used by the military). 




Table 3 

Instructional Codes 



Component taught 


Percent agreement 


Component taught 


85 


Player interaction framework: presentation 


87 


Player interaction framework: background 


99 


Player interaction framework: person-to-object 


87 


Player interaction framework: workshop 


99 


Element taught: mechanic 


93 


Element taught: control 


93 


Element taught: interface 


96 


Re-exposure: repetition 


92 


Re-exposure: elaboration 


94 


Instructional strategy: guided learning 


92 


Instructional strategy: unguided learning 


96 


Instructional method: part task 


72 


Instructional method: part whole task 


82 


Instructional method: whole task 


95 


Pre-training 


91 


Just-in-time training 


97 


Worked example 


99 


Feedback: implicit 


55 


Feedback: simple explicit 


92 


Feedback: elaborated explicit 


100 


Metacognitive support: implicit goals 


80 


Metacognitive support: explicit goals 


88 


Advance organizer 


78 


Cueing and pointers 


87 


Resource list 


86 


Task list 


79 



Table 3 lists the codes that were used in the analyses and the percent agreement for 
each code. Because 70% or greater represents an adequate level of reliability, and only the 
agreement for implicit feedback fell below 70% (it was 55%), only that item was examined 
for cause. It was discovered that one rater was over coding for feedback by coding direct 
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instruction as implicit feedback. The coder was taught the difference between instruction and 
feedback. Due to the high level of agreement (26 of 27 items exceeding 75% agreement), it 
was determined that only one rater would be needed to analyze each game of the remaining 
32 games. 



Table 4 

Relationship of Components Taught to Game Play 
Component 



Components taught 


Game play component 


Rule 


Rule 


Psychomotor skill 


Affordance 


Task 


Goal 


Tool 


Affordance 


State 


Affordance 


World space 


Game space 



As shown in Table 3, the games were analyzed for a wide range of game and 
instructional features. The first 5 rows of the table represent the elements of interest in this 
paper. Component taught refers to the components of the Game Play Model (Figure 4). There 
were six possible components that could be taught (Table 4). Note there is no component 
related to effect (see Figure 4), as effects are a reaction to what a player does, rather than 
what the player is capable of doing; that is, it is not an affordance, but rather, the result of an 
affordance. The four Player Interaction Framework items in Table 3 refer to the four ways in 
which the player can interact with information or how the information can be presented to the 
player, as illustrated in the Player Interaction Framework (Figure 3). Agreement percentages 
for game component taught and interaction method ranged from 87 percent to 99 percent, 
indicating highly reliable agreement by the two coders. 

Analyses and Results 

The analyses focused on how the six game components (Table 4) were taught. To 
ensure the two game types were similar enough in their instructional needs in order to allow 
for comparison, the percentage of instructional time devoted to each of the six game 
components was compared (Figure 5). Popular commercial games and commercial games 
utilized by the military appear to differ in the amount of instruction devoted to world space, 
constraints, and tasks. However, with the exception of game constraints (8.1% versus 3.3%), 
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the bulk of instruction on game mechanics (which is composed of state, tools, constraints, 
and psychomotor skills) is equivalent across the two game types. Since constraints represent 
the smallest portion of the game mechanics-related instruction, it appears as though the two 
game types are similar enough in how they teach game mechanics to allow for comparison. 
Therefore, the remaining analyses will be devoted to instruction of psychomotor skills, tools, 
and state-related components. 



World Space 
State 
Tool 
Constraint 
Task 
Psychomotor 

0 10 20 30 40 50 

□ Commercial ■ Military 




Figure 5. Percent of instruction devoted to game 
component by game type. 

Figure 6 shows that both game types (military and commercial) delivered equivalent 
amounts of instruction using guided learning. 




□ Commercial ■ Military 



Figure 6. Percent of guided learning. 

As can be seen in Figure 7, military games devoted more time, compared to 
commercial games, to unguided learning, particularly when teaching tools and state-related 
components; however, tools and state-related components comprised very little of the 
instruction presented in the game. 
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□ Commercial ■ Military 



Figure 7. Percent of unguided learning. 

Figures 8 through 1 1 depict the amount of instruction delivered via the four types of 
player interactions (see Figure 3). Figure 8 shows that commercial games used direct 
instruction (presentation objects) twice as often as military games, in order to teach 
psychomotor skills. 




□ Commercial ■ Military 



Figure 8. Percent of instruction delivered via 
presentation objects. 

Figure 9 shows that military games used background objects five times as often as 
commercial games to introduce state-related tools. In other words, it was up to the player to 
discover the tools because the game did not point them out. 
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State 



Tools 




Psychomotor 



P 



0 12 3 4 



□ Commercial ■ Military 



Figure 9. Percent of instruction delivered via 
background objects. 

Figure 10 shows that military games used person-to-object interactions three times as 
often, as compared to commercial games, to introduce psychomotor skills. Figures 8 and 10 
indicate that military games relied on the player choosing to interact with instructional 
objects to leam about psychomotor skills, while commercial games delivered the same type 
of instruction directly to the player, without requiring the player to seek it out. That is, 
commercial games ensured the player was exposed to the instruction while the military 
games did not. 



State 



Tools 



Psychomotor 

0 5 10 15 

□ Commercial ■ Military 

Figure 10. Percent of instruction delivered via 
person-to-object instruction 

Figure 1 1 shows that workshop objects were only used by military games for teaching 
psychomotor skills. Workshop objects were used by commercial games to teach all three 
game mechanic-related game components. 
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□ Commercial ■Military 



Figure 11. Percent of instruction delivered using 
workshop functions 



Discussion 

The analyses indicate that, while popular commercial games and commercial games 
used by the military are similar in the amount of instruction devoted to the various game play 
components (see Figure 5), they differ in how they teach those components. Both game types 
used equivalent amounts of guided learning (see Figure 5), which research has shown is 
important to learning (e.g., Kirschner, Sweller, & Clark, 2006). Commercial games used a 
greater amount of unguided learning, compared to military games, in order to teach tools 
(e.g., a weapon) and state -related components (e.g., a health bar), which supports the trial and 
error appeal of commercial video games. 

Some problems with the way military games taught how to play a game appeared when 
the methods for delivering the instruction were examined. Presentation objects directly 
deliver instruction; background objects obscure instruction; and person-to-object interactions 
require the player to actively seek instruction. Military games fall short of commercial games 
in directly teaching (presentation objects) psychomotor skills (see Figure 8), requiring instead 
that players seek out the instruction (person-to-object interactions, see Figure 10). When it 
comes to providing players with tools to monitor their condition (state), military games 
tended to blend that information into the background and relied on players awareness and 
curiosity to discover the components and how they function (see Figure 9). Finally, military 
games made very little use of workshop functions, which are tools that allow the player to 
see his or her status and resources, to manipulate those resources, and to seek out additional 
information or resources. 
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Implications 



One of the reasons popular commercial video games are popular is they are able to 
teach players how to play the game. In a market filled with competitors, these games have 
managed to engage the largest share of the market. If these games were difficult to learn, they 
would likely not have achieved that status. Research suggests the importance of integrating 
game learning with content learning, if we are to build effective learning environments that 
require using what was learned as a game feature (a game mechanic) and, at the same time, 
reduce cognitive load. The military games analyzed in this paper suggest that they used 
methods and strategies that were counter to those utilized by popular commercial games. If 
military game developers are to succeed at creating a training game that is both successful as 
a game and as a learning environment, they will need to rethink design. They will have to 
blend the delivery mechanisms of the learning domain with the delivery mechanisms of the 
game instruction, rather than the other way around. As was highlighted in this paper, 
currently, if blending does occur, the game is blended with the instructional domain, rather 
than the other way around. The blending is occurring in the wrong direction. 

The current trend that military games use background objects and person-to-object 
interactions increases the likelihood that the learner will not receive the intended instruction. 
From the research presented in the paper, the following design recommendations are 
suggested for creating military training games. 

1. Increase direct instruction (use of presentation objects) for teaching psychomotor 
skills in the game 

2. Increase direct instruction (use of presentation objects) for introducing state -related 
components 

3. Reduce the use of background objects for introducing state-related components 

4. Reduce the use of person-to-object interactions to introduce psychomotor skills 

This paper highlighted the relevance of examining game instruction in relation to the 
Player Interaction Framework (Figure 3) and the Game Play Model (Figure 4). The 
remaining 23 instruction-related elements listed in Table 3 need to be examined as well, in 
relation to the Player Interaction Framework and the Game Play Model. Examination of each 
of the remaining elements is likely to illuminate additional differences between how popular 
commercial video games teach compared to how commercial video games utilized by the 
military teach. The findings presented in this paper are one step in the direction towards the 
effective integration of game instruction and domain instruction (see Figure 1). 
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